Search for: All records

Rapid advancements in artificial intelligence (AI) necessitate changes in what AI content is taught to K-12 students. These changes will ensure that students are prepared to be smart consumers and competent creators of AI, as well as informed citizens. To meet this need, CSTA, in partnership with AI4K12, spearheaded the Identifying AI Priorities for All K-12 Students project. The project gathered experts – including teachers, researchers, administrators, and curriculum developers – to articulate priorities for AI education. This report summarizes the result of that effort. 

This report examines the similarities and differences between K-12 CS standards across seven international locations. This analysis includes background information on each of the locations included in the study, trends in content, and comparisons by topic area. 

Introduction: Learning standards are a crucial determinant of computer science (CS) education at the K-12 level, but they are not often researched despite their importance. We sought to address this gap with a mixed-methods study examining state and national K-12 CS standards. Research Question: What are the similarities and differences between state and national computer science standards? Methods: We tagged the state CS standards (n = 9695) according to their grade band/level, topic, course, and similarity to a Computer Science Teachers Association (CSTA) standard. We also analyzed the content of standards similar to CSTA standards to determine their topics, cognitive complexity, and other features. Results: We found some commonalities amidst broader diversity in approaches to organization and content across the states, relative to the CSTA standards. The content analysis showed that a common difference between state and CSTA standards is that the state standards tend to include concrete examples. We also found differences across states in how similar their standards are to CSTA standards, as well as differences in how cognitively complex the standards are. Discussion: Standards writers face many tensions and trade-offs, and this analysis shows how – in general terms – various states have chosen to manage those trade-offs in writing standards. For example, adding examples can improve clarity and specificity, but perhaps at the cost of brevity and longevity. A better understanding of the landscape of state standards can assist future standards writers, curriculum developers, and researchers in their work. 

Introduction: Recent AI advances, particularly the introduction of large language models (LLMs), have expanded the capacity to automate various tasks, including the analysis of text. This capability may be especially helpful in education research, where lack of resources often hampers the ability to perform various kinds of analyses, particularly those requiring a high level of expertise in a domain and/or a large set of textual data. For instance, we recently coded approximately 10,000 state K-12 computer science standards, requiring over 200 hours of work by subject matter experts. If LLMs are capable of completing a task such as this, the savings in human resources would be immense. Research Questions: This study explores two research questions: (1) How do LLMs compare to humans in the performance of an education research task? and (2) What do errors in LLM performance on this task suggest about current LLM capabilities and limitations? Methodology: We used a random sample of state K-12 computer science standards. We compared the output of three LLMs – ChatGPT, Llama, and Claude – to the work of human subject matter experts in coding the relationship between each state standard and a set of national K-12 standards. Specifically, the LLMs and the humans determined whether each state standard was identical to, similar to, based on, or different from the national standards and (if it was not different) which national standard it resembled. Results: Each of the LLMs identified a different national standard than the subject matter expert in about half of instances. When the LLM identified the same standard, it usually categorized the type of relationship (i.e., identical to, similar to, based on) in the same way as the human expert. However, the LLMs sometimes misidentified ‘identical’ standards. Discussion: Our results suggest that LLMs are not currently capable of matching human performance on the task of classifying learning standards. The mis-identification of some state standards as identical to national standards – when they clearly were not – is an interesting error, given that traditional computing technologies can easily identify identical text. Similarly, some of the mismatches between the LLM and human performance indicate clear errors on the part of the LLMs. However, some of the mismatches are difficult to assess, given the ambiguity inherent in this task and the potential for human error. We conclude the paper with recommendations for the use of LLMs in education research based on these findings. 

Introduction: State and national learning standards play an important role in articulating and standardizing K-12 computer science education. However, these standards have not been extensively researched, especially in terms of their cognitive complexity. Analyses of cognitive complexity, accomplished via comparison of standards to a taxonomy of learning, can provide an important data point for understanding the prevalence of higher-order versus lower-order thinking skills in a set of standards. Objective: The objective of this study is to answer the research question: How do state and national K-12 computer science standards compare in terms of their cognitive complexity? Methods: We used Bloom’s Revised Taxonomy in order to assess the cognitive complexity of a dataset consisting of state (n = 9695) computer science standards and the 2017 Computer Science Teachers Association (CSTA) standards (n = 120). To enable a quantitative comparison of the standards, we assigned numbers to the Bloom’s levels. Results: The CSTA standards had a higher average level of cognitive complexity than most states’ standards. States were more likely to have standards at the lowest Bloom’s level than the CSTA standards. There was wide variety of cognitive complexity by state and, within a state, there was variation by grade band. For the states, standards at the evaluate level were least common; in the CSTA standards, the remember level was least common. Discussion: While there are legitimate critiques of Bloom’s Revised Taxonomy, it may nonetheless be a useful tool for assessing learning standards, especially comparatively. Our results point to differences between and within state and national standards. Recognition of these differences and their implications can be leveraged by future standards writers, curriculum developers, and computing education researchers to craft standards that best meet the needs of all learners. 

Introduction: Because developing integrated computer science (CS) curriculum is a resource-intensive process, there is interest in leveraging the capabilities of AI tools, including large language models (LLMs), to streamline this task. However, given the novelty of LLMs, little is known about their ability to generate appropriate curriculum content. Research Question: How do current LLMs perform on the task of creating appropriate learning activities for integrated computer science education? Methods: We tested two LLMs (Claude 3.5 Sonnet and ChatGPT 4-o) by providing them with a subset of national learning standards for both CS and language arts and asking them to generate a high-level description of learning activities that met standards for both disciplines. Four humans rated the LLM output – using an aggregate rating approach – in terms of (1) whether it met the CS learning standard, (2) whether it met the language arts learning standard, (3) whether it was equitable, and (4) its overall quality. Results: For Claude AI, 52% of the activities met language arts standards, 64% met CS standards, and the average quality rating was middling. For ChatGPT, 75% of the activities met language arts standards, 63% met CS standards, and the average quality rating was low. Virtually all activities from both LLMs were rated as neither actively promoting nor inhibiting equitable instruction. Discussion: Our results suggest that LLMs are not (yet) able to create appropriate learning activities from learning standards. The activities were generally not usable by classroom teachers without further elaboration and/or modification. There were also grammatical errors in the output, something not common with LLM-produced text. Further, standards in one or both disciplines were often not addressed, and the quality of the activities was often low. We conclude with recommendations for the use of LLMs in curriculum development in light of these findings. 

There is a need to analyze state computer science standards to determine their cognitive complexity and alignment across grades. However, due to the recency of these standards, there is very little research on the topic, including the use of various educational taxonomies as analysis tools. The purpose of this paper is to answer the question, How do Bloom’s Revised and the SOLO taxonomies compare in their analysis of computer science standards? We categorized state CS standards according to their level in Bloom’s Revised Taxonomy and the SOLO taxonomy. Analyzing state CS standards using the Bloom’s or using the SOLO taxonomy produces wide areas of agreement but also some differences that might be important in various use cases, such as aligning standards across grade levels or determining whether a standard addresses a higher-order thinking skill. 

This report examines the similarities and differences between state-adopted K-12 CS standards and the 2017 CSTA K-12 Standards. This analysis includes basic information about the standards (such as counts by state and level) and their cognitive complexity, as well as more detailed information about their relationship to the CSTA standards. 

In recent years, eight states have adopted a graduation requirement in computer science (CS), and other states are considering similar requirements. Due to the recency of these requirements, little is known about student and teacher perceptions of course(s) that fulfill the requirement and their content. This project seeks to answer the question, What are the perceptions of students who are studying CS beyond high school and CS teachers of a high school CS requirement and its content? We used a mixed methods approach that included interview transcripts from students who took CS coursework in high school and are currently studying it in college (n = 9). We also used quantitative data from a survey of CS teachers (n = 2, 238) that asked for their perceptions of a CS graduation requirement. Most of the students felt that CS should be required in high school, and there was a wide variety of sentiment regarding what content should be included in such a course. For the high school teachers, about 85% felt that CS should be required. It is perhaps not surprising that most students who studied CS in college valued it at the high school level and thus supported a graduation requirement. What is more interesting is the diversity of content that they felt should belong in such a course. These findings serve as an important consideration for those implementing a CS graduation requirement. 

In the United States, state learning standards guide curriculum, assessment, teacher certification, and other key drivers of the student learning experience. Investigating standards allows us to answer a lot of big questions about the field of K-12 computer science (CS) education. Our team has created a dataset of state-level K-12 CS standards for all US states that currently have such standards (n = 42). This dataset was created by CS subject matter experts, who - for each of the approximately 10,000 state CS standards - manually tagged its assigned grade level/band, category/topic, and, if applicable, which CSTA standard it is identical or similar to. We also determined the standards' cognitive complexity using Bloom's Revised Taxonomy. Using the dataset, we were able to analyze each state's CS standards using a variety of metrics and approaches. To our knowledge, this is the first comprehensive, publicly available dataset of state CS standards that includes the factors mentioned previously. We believe that this dataset will be useful to other CS education researchers, including those who want to better understand the state and national landscape of K-12 CS education in the US, the characteristics of CS learning standards, the coverage of particular CS topics (e.g., cybersecurity, AI), and many other topics. In this lightning talk, we will introduce the dataset's features as well as some tools that we have developed (e.g., to determine a standard's Bloom's level) that may be useful to others who use the dataset.